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CROSS-REFERENCE TO RELATED APPLICATIONS 

This appUcation is related to appUcation Attorney Docket No. AP102HO, 
entitled "Image Artifact Removal Technique for LCP," filed herewith in names of 
inventors Joseph Shu, Jau-Yuen Chen and Tsung-Nan Lin and assigned to the 
10 assignee of this application. The contents of the above-identified related application 
are incorporated by reference herein. 

Ji BACKGROUND OF THE INVENTION 

Field of the Invention 

\^ This invention relates to a memory efficient method and apparatus for 

iJS removing image artifacts including moire from screen images, particularly images 
% generated by liquid crystal projectors (LCPs). The invention also relates to a 
U program of instructions for implementing various aspects of the image artifact 
removal technique. 

CI Description of the Related Art 

20 An image that contains screens, which are composed of periodic lines or dot 

patterns, sometimes exhibits artifacts including moire artifacts when projected or 
displayed. Moire artifacts may result whenever two geometrically-regular patterns 
are superimposed and often manifests itself as a ripple-like pattern in the image 
representations. Such artifacts degrade image quality and are therefore 

25 undesirable. 

In the case of a projected image, the image-generating projection device (e.g., 
an LCP) may employ a technique known as a "keystone correction" which alters the 
shape of the projected image to compensate for the angle of projection. While the 
keystone correction improves certain characteristics of the projected image, it has a 
30 disadvantage in that it is usually not able to maintain equal spacing between 
screen fine/dot patterns. Consequently, the unequal spacing at different locations 
in the image results in moire artifacts. 
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While low pass filtering may be used to remove the moire artifacts, global 
application of a low pass filter (LPF) tends to cause blurring in text and non-screen 
regions in the displayed or projected image. That is, if an LPF is applied uniformly 
to the image, the degree of filtering required to achieve an acceptable reduction in 
5 moire usually results in an unacceptable reduction in resolution. 

Thus, there is a need for an effective image artifact removal technique that 
smoothes screen regions to remove moire while maintaining sharpness in non- 
screen regions and that is particularly well suited for projected images. 

OBJECTS AND SUMMARY OF THE INVENTION 

10 Objects of the Invention 

Therefore, it is an object of the present invention to overcome the 

= aforementioned problems by providing such a memory efficient image artifact 

%A removal technique. 

m It is another object of this invention to provide a memory efficient image 

nt5 artifact removal technique that identifies and segments screen and non-screen 

in image regions and treats the two regions separately to remove moire from the 

I- screen regions while maintaining sharpness in the non-screen regions. 

Summary of the Invention 

According to one aspect of this invention, a memory efficient method for 
1=^20 removing image artifacts from an image representation is provided. Such method 
comprises the steps of (a) obtaining a pixel representation of the image; (b) 
classifying each pixel in the image as a screen or non-screen pixel; (c) examining 
pixels in a predetermined surrounding area of each pixel to check the classification 
of that pixel as determined in step (b); and (d) selectively applying a low pass filter 
25 to pixels in the image, such that, when the low pass filter is applied, one or more 
pixels covered by the low pass filter are respectively replaced by one or more other 
pixels covered by the low pass filter based on the examining in step (c). 

Various preferred features of the method are set forth below. 

The classifying step (b) comprises applying a first mask of a predetermined 
30 size centered on the pixel being classified to determine if the center pixel is in an 
area having a predetermined periodic pattern. 

The first mask is divided into a plurahty of overlapping areas, the center 
pixel being in each of the first mask areas. 
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The examining step (c) comprises applying a second mask of a predetermined 
size centered on the pixel being checked. 

The second mask is divided into a plurality of overlapping areas, the center 
pixel being in each of the second mask areas. 

5 The predetermined periodic pattern is a periodic line or dot pattern having a 

period of 2 or 3. 

The selectively applying step (e) comprises selectively applying the low pass 
filter based on which of the plurality of second mask areas contains screen pixels. 

The above-described method may further comprise the steps of (f) 
10 determining a feature indicator for at least one portion of the image; and (g) 
adaptively sharpening or softening the at least one portion of the image based on 
the determined feature indicator. 

Another aspect of the invention involves an apparatus for removing image 
artifacts from a representation of an image. Such apparatus comprises a device for 
i;H5 obtaining a pixel representation of the image; a screen pixel identifier, in 
communication with the device, for classifying each pixel in the image as a screen or 
I d non-screen pixel; a screen region verifier, in communication with the screen pixel 
'l^^ identifier, for examining pixels in a predetermined surrounding area of each pixel to 
check the classification of that pixel as determined by screen pixel identifier; and a 
1^0 low pass filter, in communication with the screen region verifier, that is selectively 
^•J applied to the pixels in the image, such that, when the low pass filter is applied, one 
or more pixels covered by the low pass filter are respectively replaced by one or 
more other pixels covered by the low pass filter based on the examining. 

Various preferred features of the apparatus are set forth below. 

25 The screen pixel identifier comprises a first mask of a predetermined size 

that is applied by centering the first mask on the pixel being classified to determine 
if the center pixel is in an area having a predetermined periodic pattern. 

The first mask is divided into a pluraUty of overlapping areas, the center 
pixel being in each of the first mask areas. 

30 The screen region verifier comprises a second mask of a predetermined size 

that is applied by centering the second mask on the pixel being checked. 

The second mask is divided into a plurality of overlapping areas, the center 
pixel being in each of the second mask areas. 
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The predetermined periodic pattern is a periodic line or dot pattern having a 
period of 2 or 3. 

The low pass filter is selectively applied based on which of the plurality of 
second mask areas contains screen pixels. 

5 The apparatus described above may further comprise a frequency classifier 

that determines a feature indicator for at least one portion of the image; and an 
image processor for adaptively sharpening or softening the at least one portion of 
the image based on the determined feature indicator. 

In accordance with further aspects of the invention, the above-described 
10 method or steps thereof may be embodied in a program of instructions (e.g., 
software) which may be stored on, or conveyed to, a computer or other processor- 
controlled device for execution. Alternatively, the method or steps thereof may be 
1- implemented using hardware or a combination of software and hardware. 

I Other objects and attainments together with a fuller understanding of the 

;-^5 invention will become apparent and appreciated by referring to the following 
description and claims taken in conjunction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

i^: In the drawings wherein Uke reference symbols refer to like parts: 

^ Fig. 1 is a block diagram illustrating major components in an image 

ijo projection system that is suitable for carrying out various aspects of the present 
invention; 

Fig. 2 is a block diagram illustrating components in a typical computer 
system that is suitable for carrying out various aspects of the invention; 

Fig. 3 is a functional block diagram showing the process flow and the 
25 relationship between the keystone image artifact removal (KIAR), the adaptive 
field-based video enhancement (AFBVE), and the Keystone Morphing operations; 

Fig. 4 is a functional block diagram of the KIAR technique according to 
embodiments of the invention; 

Figs. 5 A and 5B are schematic diagrams of screen pixel identifier (SFI) 
30 masks according to embodiments of the invention; 

Fig. 6 is a schematic diagram of a screen region verifier (SRV) mask 
according to embodiments of the invention; 
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Fig. 7 is a schematic representation of a 3 x 3 Gaussian LPF kernel used in 
embodiments of the invention; 

Fig. 8 is a schematic representation of the numbering of the pixels covered by 
the LPF kernel; 

5 Fig. 9 is a functional block diagram of the AFBVE technique according to 

embodiments of the invention; 

Fig. 10 is a graphical representation of the value of a feature indicator 
produced by a frequency classifier of the AFBVE operation as a function of Sobel 
edge detector output; 

10 Fig. IIA is a graphical representation of an interlaced input image; and 

Fig. IIB is a schematic representation showing how a filter kernel, such the 
^ LPF kernel shown in Fig. 7, is applied to interlaced image data. 

y DESCRIPTION OF THE PREFERRED EMBODIMENTS 

ill Fig. 1 illustrates components in a typical image projection system 10 in 

i l5 which the techniques of the present invention can be employed. In the illustrated 
embodiment, a computer system 12, stores digital images that are electrically 
^3 transmitted along a suitable transmission path 14 to a projecting device 16, such as 
Mk a liquid crystal projector (LCP) which projects such images onto a screen, wall or 
other display area. As will be appreciated by those skilled in the art, Fig. 1 
represents just one of many alternatives available for obtaining digital images and 
projecting them. The present invention concerns certain processing applied to the 
digital images before they are displayed or projected, and for the purpose of this 
invention, it is not important how or where the images are stored or digitized. 
Digital images to be processed in accordance with the invention may be obtained 
25 from scanners, digital cameras, etc., stored in a memory, and transmitted to the 
projecting device. Such digital images may also be computer generated. Moreover, 
the special processing which is the subject of the present invention may occur either 
in the projecting device itself, or in a computer or other device capable of performing 
the processing of the invention prior to transmission of the processed image data to 
30 the projecting device. 

In the case where digitized images are obtained, stored and processed by 
computer 12, such computer may be of any suitable type, including a personal 
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computer or workstation. As illustrated in Fig. 2, the computer typically includes a 
central processing unit (CPU) 21 that provides computing resources and controls 
the computer. CPU 21 may be implemented with a microprocessor or the like, and 
may also include a graphics processor and/or a floating point coprocessor for 
5 mathematical computations. Computer 12 further includes system memory 22 
which may be in the form of random-access memory (RAM) and read-only memory 
(ROM). 

A number of controllers and peripheral devices are also provided, as shown in 
Fig. 2. Input controllers 23 represents an interface to one or more input devices 24, 
10 such as a keyboard, mouse or stylus. Computer system 12 may also have input 
controllers for connecting an input device such as a scanner and/or digital camera 
from which digital images may be obtained. A storage controller 25 is an interface to 
2 a storage device 26 that includes a storage medium such as magnetic tape or disk, 
gi or an optical medium that may be used to record programs of instructions for 
p|5 operating systems, utilities and applications which may include embodiments of 
programs that implement various aspects of the present invention. Storage device 
Q 26 may also be used to store image data to be processed in accordance with the 
invention. Output controllers 27 provide interfaces to output devices 28 such as a 
=5 display device which may be a cathode ray tube (CRT) or thin film transistor (TFT) 
^'20 display. An output controller is also provided for connecting projecting device 16 to 
computer 12. Communications controller 29 interfaces with communication device 31 
which may be a modem or other network connection. Programs that implement 
various aspects of this invention and/or processed or to be processed image data may 
be transmitted to computer 12 from a remote location (e.g., a server) over a network. 

25 In the illustrated embodiment, all major system components connect to bus 32 

which may represent more than one physical bus. For example, some personal 
computers incorporate only a so-called Industry Standard Architecture (ISA) bus. 
Other computers incorporate an ISA bus as well as a higher bandwidth bus. 

Fig. 3 illustrates a processor 100, which may be embodied in computer 12, in 
30 the projecting device 16, or other suitable device, for performing various functions, 
including keystone image artifact removal (KIAR) 35, adaptive field-based video 
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enhancement (AFBVE) 36, which KIAR is designed to support, and keystone 
morphing 37. The relationship between KIAR, AFBVE and keystone morphing is 
illustrated in the figure. The input image (I/P) is transmitted to the KIAR block 
where it is processed in accordance with the KIAR technique. The KIAR processed 
5 image is then transmitted to the AFBVE block where it may be further processed in 
accordance with the AFBVE technique. Keystone morphing may then be applied to 
AFBVE processed image to generate an output image (O/P) that is displayed by 
LCP 16. 

The KIAR, AFBVE and keystone morphing operations may be implemented 
10 in processor 100 in a variety of ways including by software, hardware, or 

combination thereof. Such software, for example, may be stored on computer 12 in 
'2 system memory 22 or in storage device 26, and fetched by CPU 21 for execution. 

More broadly, such software may be conveyed by any of a variety of machine- 
ip; readable medium including magnetic tape or disk, optical disc, signals transmitted 
145 through network paths including the Internet, as well as other suitable carrier 

signals throughout the electromagnetic spectrum including infrared signals. 
O Processor 100 may also be implemented with discrete logic circuits, one or more 
u application specific integrated circuits (ASICs), digital signal processors, program- 
J'Sj controlled processors, or the hke. 

20 Fig. 4 is a functional block diagram of the KIAR technique in accordance with 

embodiments of the invention. In one embodiment, KIAR 35 is implemented with a 
screen pixel identifier (SPI) 41, a screen region verifier (SRV) 42 and a filter 43 such 
as a low pass filter (LPF). The process flow is as shown in the figure. SPI 41 
examines each pixel of a digital image input to KIAR block 35 and classifies that 

25 pixel as either a screen or non-screen pixel. For each pixel, SRV 42 then examines 
surrounding area pixels to check if the initial classification was correct and to 
reclassify any pixels misclassified by SPI 41, A LPF is then applied to screen region 
pixels, as described below, while keeping the non-screen region pixels unchanged. 

SPI 41 employs a one-dimensional mask, one embodiment of which is 
80 schematically illustrated in Fig. 5A and another of which is shown in Fig. 5B, to 
perform its classification function. Masks 51a and 51b each have a size of 25 x 1 
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pixels and is centered on the pixel currently being examined. The mask elements 
52a and 52b are spaced so as to check if the center pixel is in a periodic line or dot 
pattern area, where the period or distance (dist) between pixels corresponding to 
adjacent mask elements is equal to some predetermined number. In the mask of 
5 Fig. 5A, elements 52a are spaced so that the predetermined number is 2. That is, 
there is one pixel between each pixel that corresponds to a mask element and that 
is examined. In the mask of Fig. 5B, elements 52b are spaced so that the 
predetermined number is 3. That is, there are two pixels between each pixel that 
corresponds to a mask element and that is examined. 

10 As illustrated in Figs. 5A and 5B, each mask 51a and 51b is divided into 

overlapping areas (1) and (2) with one pixel overlap in the center. Such division is 
done to facilitate the identification of screen pixels on borders and in corners. Each 
y area (1) and (2) corresponds to 5 or 7 pixel locations, with spacing denoted by "dist" 
ijl as 2 or 3 pixels. Mask 51a examines 7 pixels in each area with a spacing of 2 pixels, 
145 while mask 51b examines 5 pixels in each area with a spacing of 3 pixels. 

In one embodiment, if all examined pixels in a particular area have the same 
'[Z color values (e.g., RGB values), then that area is considered to be in a screen area 
J^J with a predetermined screen pattern period corresponding to the SPI mask used. 
;3 Other pixel attributes, such as luminosity or chrominance, may also be examined 
20 for patterns corresponding to the predetermined pattern. If either area is classified 
as being in a screen area, then the center pixel is considered a screen pixel. 

After processing with the SPI mask, any pixel misclassifications are corrected 
by SRV 42 which also employs a mask. When the SPI mask is 25 x 1, as shown in 
Figs. 5A and 5B, the SRV mask 61 is preferably 25 x 3 pixels and is divided into 

25 quadrants denoted [1], [2], [3] and [4], as illustrated in Fig. 6. Each quadrant is 
preferably 13 x 2 pixels with one horizontal and vertical hne overlap between 
adjacent quadrants. That is, the right-most column in quadrant [1] overlaps with 
the left-most column in quadrant [2], and the lower-most row of quadrant [1] 
overlaps with the upper-most row of quadrant [3]. Similarly, the left-most column 

30 in quadrant [4] overlaps with the right-most column in quadrant [3], and the upper- 
most row in quadrant [4] overlaps with the lower-most row of quadrant [2]. 
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Dividing the SRV mask 61 into quadrants allows the SRV to perform accurately 
both inside screen regions and at the borders and corners of such regions. 

A filter kernel of LPF 43 is applied based on the processing with the SRV 
mask. The filter kernel is preferably a 3 x 3 Gaussian LPF kernel whose 
5 coefficients are shown in Fig. 7, The manner in which the LPF kernel is applied 
depends on which combination of quadrants of the SRV mask 61 contain screen 
pixels. If all of the pixels in each of the four quadrants of SRV mask 61 are screen 
pixels, then the LPF kernel is applied to smooth the center pixel, which is the 
darkened pixel in Fig. 6. However, if only some quadrants contain screen pixels, 

10 the corresponding pixels are replaced by "mirror" pixels, as described below with 
reference to Fig. 8 which denotes the numbering of the pixels covered by the LPF 

:d: kernel. 

y' If only quadrants [1], [2] and [3] contain screen pixels, pixel 9 is replaced by 

pixel 1. If only quadrants [1], [2] and [4] contain screen pixels, pixel 7 is replaced by 

il5 pixel 3. If only quadrants [1], [3] and [4] contain screen pixels, pixel 3 is replaced by 
pixel 7. If only quadrants [2], [3], and [4] contain screen pixels, pixel 1 is replaced 

j;^ by pixel 9. If only quadrants [1] and [2] contain screen pixels, pixels 7, 8, and 9 are 
replaced by pixels 1, 2, and 3, respectively. If only quadrants [1] and [3] contain 

Q screen pixels, pixels 3, 6, and 9 are replaced by pixels 1, 4, and 7, respectively. If 

20 only quadrants [2] and [4] contain screen pixels, pixels 1, 4, and 7 are replaced by 
pixels 3, 6, and 9, respectively. If only quadrants [3] and [4] contain screen pixels, 
pixels 1, 2, and 3 are replaced by pixels 7, 8, and 9, respectively. If only quadrant [1] 
contains screen pixels, pixels 3, 6, 9, 8, and 7 are replaced by pixels 1, 2, 1, 4, and 1, 
respectively. If only quadrant [2] contains screen pixels, pixels 1, 4, 7, 8, and 9 are 

25 replaced by pixels 3, 2, 3, 6, and 3, respectively. If only quadrant [3] contains screen 
pixels, pixels 1, 2, 3, 6, and 9 are replaced by pixels 7, 4, 7, 8, and 7, respectively. If 
only quadrant [4] contains screen pixels, pixels 3, 2, 1, 4, and 7 are replaced by 
pixels 9, 6, 9, 8, and 9, respectively. In all other cases, the pixel being examined is 
left unchanged. That is, the LPF is not applied. 

30 For a pixel (i, j), the LPF process is as follows: 

li/p (i, j) = input image of KIAR operation 
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Iipf (i J) - 3 X 3 Gaussian LPF 
Ihpf (i, j) = li/p (i, j) - Iipf (i, j) 

lo/p (i, j) = output image of KIAR operation = Iipf (i, j) + (1 + sharpening factor) 
Ihpf (i, j) 

5 While the sharpening factor may be experimentally determined for a given LCP 
system based on the system's hardware characteristics, the inventors have 
experimentally determined that a value for the term (1 + sharpening factor) — 1/8. 
Such a value reduced moire artifacts in screen areas under all keystone operations 
on a test image. For non-screen areas, the output image is made to be the same as 
10 the input image to maintain non-screen areas of the image. 

^3 As previously noted, the KIAR technique may be employed with another 

y technique known as Adaptive Field-Based Video Enhancement (AFBVE). In this 

case, the KIAR processed image is passed to the AFBVE block for further smoothing 
]Z or sharpening, as determined by the AFBVE technique. AFBVE is an adaptive 
^JS process that is used to smooth or sharpen edges or boundaries, as needed, to remove 

blurring caused by interlaced video input, which is the separate input to the LCP of 
i odd and even image pixel lines. With interlaced video input, usually the odd pixel 

lines are input first, followed by the even pixel Hnes. Either way, such an input 

results in each video frame being comprised of two fields: an odd line field and an 
20 even line field. These interaction between these two fields tends to cause blurring 

in the resulting image. AFBVE is designed to correct this type of blurring. 

Fig. 9 is a functional block diagram of the AFBVE technique which 
adaptively applies sharpening and smoothing to the digital output of the KIAR 
block 35. The output of KIAR 35 forms the input image, which is interlaced or 
25 progressive data, to AFBVE 36. This input image is transmitted to LPF 91, a first 
summing function 92, and a frequency classifier 93. 

LPF 91 low-pass filters the input image to obtain a smooth (i.e., softened 
image). LPF 91 preferably uses the same 3x3 Gaussian kernel illustrated in Fig. 
7. The smoothened image is subtracted from the input image to obtain the high 
30 pass frequency components of the image which forms the other input to first 
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summing function 92. Element 92 combines the input image and the high pass 
frequency components and transmits the result to a scale factor determiner 94 
which generates a scaling factor for magnifying the high frequency components of 
the image. A scaling factor of 1 means that the output image of the AFBVE 
5 function is to be the same as the function's input image; a scaling factor greater 
than 1 means the output image is to be sharpened; and a scaling factor of less than 
1 means the output image is to be softened. The inventors have experimentally 
determined a preferred scaling factor of 1 + 20/32 = 52/32. 

Rather than simply magnifying the high frequency components by the scaling 
10 factor, and then adding them on to the smoothed image to produce the output (0/P) 

image, the frequency classifier 93 of the present invention produces a feature 
'Z: indicator, denoted by in Fig. 9, to modulate the scaling factor to achieve adaptive 
^ sharpening and softening enhancement. Frequency classifier 93 determines which 
ffl image areas are to be sharpened and which are to be softened and also determines 
lis the magnitude of the sharpening or softening factor to be applied at each pixel 

location in the image. That is, frequency classifier 93 detects image spatial 
■^'^ frequency information in order to separate primary edges from noisy micro-edges 
M (or non-edges) to produce the feature indicator, /. 

^3 Specifically, frequency classifier 93 uses 3x3 Sobel edge detectors to detect 

20 spatial frequency components of the image and classify each component into one of 
the three regions illustrated in Fig. 10. The three regions are: a softening region 
which is below a predetermined lower threshold frequency (thr_low), a sharpening 
region above a predetermined upper threshold (thr_high), and a transition region 
between the softening and sharpening regions. The transition region provides a 
25 smooth transition between softening and sharpening enhancement. As shown in 
Fig. 10, / is linear in the transition region to avoid discontinuity artifacts between 
the softening and sharpening regions. 

As shown in Fig. 10, feature indicator, /, is a function of the output of the 
Sobel edge detectors, which has a range of 0 — 255. The thr_low and thr_high will 
30 vary depending on the type of image, personal visual preferences and system 
hardware characteristics. Such thresholds can be experimentally determined. 
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However, the inventors have determined that for many images a thr__low of about 
20 and thr_high of about 40 yields good results. 

The feature indicator /, which is number from 0 to 1 (expressed as a 
percentage between 0 and 100% in Fig. 10), is input to a multiplier 95 where it is 
5 multiplied with the scaling factor of scale factor determiner 94 to produce a 
modulated scaling factor which is applied by summer 96 to the smoothed image 
produced by LPF 91. A modulated scaling factor of 1 means that no change is 
applied to output of LPF 91; a scaling factor greater than 1 means that such output 
is sharpened; and a scaling factor of less than 1 means that such output is softened 
10 or smoothed. The resulting image from summer 96 is the output (O/P) image of the 
AFBVE process. 

Fig. IIA is a graphical representation of an interlaced video image which 
ijl may be input to an LCP. As previously discussed, in an interlaced image format, 
'{\ each frame contains two fields: a first field which contains, say, all of the odd scan 
!S5 lines and a second field which contains, say, all of the even scan lines. Each image 
data field is processed separately, for example, the first field is processed followed 
by the second field. Thus, in the filtering process, a filter kernel is applied to 
elements from either the first or the second fields, as schematically illustrated in 
3 Fig. IIB. It should be noted that the interlaced video format is only one type of 
20 video input for an LCP. A progressive format, an image of which has a line-by-line 
raster format, may also be used. 

As the foregoing description demonstrates, the present invention provides a 
technique (KIAR) for smoothing non-screen image regions to remove moire 
therefrom while maintaining sharpness in the non-screen regions. The KIAR 
25 process may be followed by an AFBVE process, each of which may be conveniently 
implemented in a personal computer or other processing device using software, 
hardware, or combination thereof. 

With these implementation alternatives in mind, it is to be understood that 
the block and flow diagrams show the performance of certain specified functions 
30 and relationships thereof. The boundaries of these functional blocks have been 
arbitrarily defined herein for convenience of description. Alternate boundaries may 
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be defined so long as the specified functions are performed and relationships 
therebetween are appropriately maintained. The diagrams and accompanying 
description provide the functional information one skilled in the art would require 
to fabricate circuits or to write software code to perform the processing required. 

5 While the invention has been described in conjunction with several specific 

embodiments, many further alternatives, modifications, variations and applications 
will be apparent to those skilled in the art that in light of the foregoing description. 
Thus, the invention described herein is intended to embrace all such alternatives, 
modifications, variations and apphcations as may fall within the spirit and scope of 
10 the appended claims. 
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